Accuracy and reproducibility of forensic tire examination decisions.

Tire impression evidence can be a valuable tool during a crime scene investigation-it can link vehicles to scenes or secondary locations, and reveal information about the series of events surrounding a crime. The interpretation of tire impression evidence relies on the expertise of forensic tire examiners. To date, there have not been any studies published that empirically evaluate the accuracy and reproducibility of decisions made by tire impression examiners. This paper presents the results of a study in which 17 tire impression examiners and trainees conducted 238 comparisons on 77 distinct questioned impression-known tire comparison sets (QKsets). This study was conducted digitally and addressed examinations based solely upon the characteristics of the tire impression images provided. The quality and characteristics of the impressions were selected to be broadly representative of those encountered in casework. Participants reported their decisions using a multi-level conclusion scale: 68% of responses were class associations (Association of Class Characteristics or Limited Association of Class), 21% were definitive decisions (ID or Exclusion), 8% were probable decisions (High Degree of Association or Indications of Non-Association), and 3% were neutral responses (Not Suitable or Inconclusive). Although class associations were the most reported response type, when definitive decisions were reported, they were often correct: 96% of IDs and 89% of Exclusions were consistent with ground truth regarding the source of the known tire in the QKset. Overall, we observed 4 erroneous definitive decisions (3 Exclusions on mated QKsets; 1 ID on a nonmated QKset) and 1 incorrect probable decision (Indications of Non-Association on a mated QKset). Decision rates were notably associated with both quality (lower quality questioned impressions were more likely to result in class associations) and dimensionality (2D questioned impressions were more likely to result in definitive decisions), which were correlated factors. Although the study size limits the precision of the measured rates, the results of this study remain valuable to the forensic science and legal communities and provide empirical data regarding examiner performance for a discipline that previously did not have any such estimates.

Variation in assessments of suitability and number of contributors for DNA mixtures.

The interpretation of a DNA mixture (a sample that contains DNA from two or more people) depends on a laboratory/analyst's assessment of the suitability of the sample for comparison/analysis, and an assessment of the number of contributors (NoC) present in the sample. In this study, 134 participants from 67 forensic laboratories provided a total of 2272 assessments of 29 DNA mixtures (provided as electropherograms). The laboratories' responses were evaluated in terms of the variability of suitability assessments, and the accuracy and variability of NoC assessments. Policies and procedures related to suitability and NoC varied notably among labs. We observed notable variation in whether labs would assess a given mixture as suitable or not, predominantly due to differences in lab policies: if two labs following their standard operating procedures (SOPs) were given the same mixture, they agreed on whether the mixture was suitable for comparison 66% of the time. Differences in suitability assessments have a direct effect on variability in interpretations among labs, since mixtures assessed as not suitable would not result in reported interpretations. For labs following their SOPs, 79% of assessments of NoC were correct. When two different labs provided NoC responses, 63% of the time both labs were correct, and 7% of the time both labs were incorrect. Incorrect NoC assessments have been shown to affect statistical analyses in some cases, but do not necessarily imply inaccurate interpretations or conclusions. Most incorrect NoC estimates were overestimates, which previous research has shown have less of an effect on likelihood ratios than underestimates.

DNAmix 2021: Laboratory policies, procedures, and casework scenarios summary and dataset.

DNAmix 2021 is a large-scale study conducted to evaluate the extent of consistency and variation among forensic laboratories in the interpretation of DNA mixtures, and to assess the effects of various potential sources of variability. This study utilized a multi-phasic approach designed to collect information about participating laboratories, laboratory policies, and their standard operating procedures (SOPs). It also characterizes the degree of variation in assessments of suitability and number of contributors as well as in comparisons and statistical analyses of DNA mixture profiles. This paper specifically details the study design and the data collected in the first two phases of the study: the Policies & Procedures (P&P) Questionnaire and the Casework Scenarios Questionnaire (CSQ). We report on the variation in policies and SOPs for 86 forensic laboratories-including information about their DNA workflows, systems, and type of statistics reported. We also provide details regarding various case-scenario specific decisions and the nature of mixture casework for 83 forensic laboratories. The data discussed in this article provide insight into the state of the field for forensic DNA mixture interpretation policies and SOPs at the time of the study (2021-2022).

Spatial frequency of randomly acquired characteristics on outsoles.

The highest levels of source association in forensic footwear comparison rely on the agreement between randomly acquired characteristics (RACs) identified on questioned and exemplar test impressions. These features are presumed to be randomly acquired and independent. However, independent acquisition does not necessarily mean these features will be uniformly distributed across an outsole. The aim of this research was to determine if the distribution of RACs in a research dataset could be described by an inhomogeneous Poisson point process based on tread contact and wear. To achieve this goal, RAC spatial frequency from an empirical dataset of shoes was compared against simulated and modeled data assuming a Poisson point process. Deviations in count between the empirical and simulated/modeled predictions were examined using a Poisson rate test and Moran's I. Results indicate that RAC frequency over 67%-79% of an outsole can be reasonably well explained as a Poisson point process or by a Poisson generalized linear regression model (non-spatial GLM) with tread contact as a predictor. Moreover, if the predictor is extended to include both tread contact and wear, RAC counts over 84% of the spatial locations on an outsole are well-explained (although autocorrelation persists). Overall, results indicate that RACs are not uniformly distributed in this dataset, most likely because the factors that dictate RAC development (friction, gait, etc.) are not uniformly distributed. Although this observation in no way negates the use of RACs in forming source associations, the value of a correspondence can differ depending on its spatial location.

Accuracy, reproducibility, and repeatability of forensic footwear examiner decisions.

The interpretation of footwear evidence relies on the expertise of forensic footwear examiners. Here we report on the largest study to date of the accuracy, reproducibility (inter-examiner variation), and repeatability (intra-examiner variation) of footwear examiners' decisions. In this study, 84 practicing footwear examiners each conducted up to 100 comparisons between questioned footwear impressions (provided as photographs and digital images) and known footwear (provided as photographs, transparent test impressions, and digital images), resulting in a total of 6610 comparisons. The quality and characteristics of the impressions were selected to be broadly representative of those encountered in casework. A multilevel conclusion scale was used: 40% of responses were definitive conclusions (identification or exclusion), 14% probable conclusions (high degree of association or indications of non-association), 40% class associations (association of class characteristics or limited association of class characteristics), and 6% neutral conclusions (inconclusive or not suitable). On nonmated comparisons, 0.2% of conclusions were erroneous identifications (false positives), and 1.4% were incorrect responses of "high degree of association." The majority of erroneous identifications were made by a single participant. On mated comparisons, 6.0% of conclusions were erroneous exclusions (false negatives), and 1.8% were incorrect responses of "indications of non-association." Erroneous conclusions were sometimes reproduced by different examiners, but rarely repeated by the same examiner-1.1% of erroneous identifications were reproduced (none were repeated) and 19.9% of erroneous exclusions were reproduced (just one was repeated). Examiners' assessments of whether a questioned impression was suitable for comparison were notably inconsistent and may benefit from standardization. Rates of correct definitive conclusions are directly associated with the quality of the questioned impression and the extent of class similarities/differences between the questioned impression and known footwear.

Accuracy and reliability of forensic handwriting comparisons.

Forensic handwriting examination involves the comparison of writing samples by forensic document examiners (FDEs) to determine whether or not they were written by the same person. Here we report the results of a large-scale study conducted to assess the accuracy and reliability of handwriting comparison conclusions. Eighty-six practicing FDEs each conducted up to 100 handwriting comparisons, resulting in 7,196 conclusions on 180 distinct comparison sets, using a five-level conclusion scale. Erroneous "written by" conclusions (false positives) were reached in 3.1% of the nonmated comparisons, while 1.1% of the mated comparisons yielded erroneous "not written by" conclusions (false negatives). False positive rates were markedly higher for nonmated samples written by twins (8.7%) compared to nontwins (2.5%). Notable associations between training and performance were observed: FDEs with less than 2 y of formal training generally had higher error rates, but they also had higher true positive and true negative rates because they tended to provide more definitive conclusions; FDEs with at least 2 y of formal training were less likely to make definitive conclusions, but those definitive conclusions they made were more likely to be correct (higher positive predictive and negative predictive values). We did not observe any association between writing style (cursive vs. printing) and rates of errors or incorrect conclusions. This report also provides details on the repeatability and reproducibility of conclusions, and reports how conclusions are affected by the quantity of writing and the similarity of content.

Accuracy and reproducibility of conclusions by forensic bloodstain pattern analysts.

Although the analysis of bloodstain pattern evidence left at crime scenes relies on the expert opinions of bloodstain pattern analysts, the accuracy and reproducibility of these conclusions have never been rigorously evaluated at a large scale. We investigated conclusions made by 75 practicing bloodstain pattern analysts on 192 bloodstain patterns selected to be broadly representative of operational casework, resulting in 33,005 responses to prompts and 1760 short text responses. Our results show that conclusions were often erroneous and often contradicted other analysts. On samples with known causes, 11.2% of responses were erroneous. The results show limited reproducibility of conclusions: 7.8% of responses contradicted other analysts. The disagreements with respect to the meaning and usage of BPA terminology and classifications suggest a need for improved standards. Both semantic differences and contradictory interpretations contributed to errors and disagreements, which could have serious implications if they occurred in casework.

Forensic Footwear Reliability: Part III-Positive Predictive Value, Error Rates, and Inter-Rater Reliability.

Over the course of 19 months, West Virginia University collected reports from 70 footwear experts, each performing 12 questioned-test comparisons, resulting in a dataset that includes more than 1000 examiner attributes (education, training, certification status, etc.), 3500 impression features identified and evaluated (clarity, totality, and similarity), and 840 source conclusions. The results were used to estimate the performance of forensic footwear examiners in the United States, including error rates, predictive value (PV), and measures of inter-rater reliability (IRR). For the dataset and mate-prevalence (31.5%) used in this study, results indicate correct predictive value varies from 94.5% for exclusions, 85.0% for identifications, and between 70.1% and 65.2% for limited associations and association of class, respectively (with all other conclusions producing PVs between these extremes). After data transformation based on ground truth, the case study materials show a false-positive rate of 0.48%, a false-negative rate of 15.6%, a (correct) positive predictive value of 98.8%, and a (correct) negative predictive value of 93.3%. In addition to error rates and PVs, inter-rater reliability was likewise computed to describe examiner reproducibility; results indicate a Gwet AC2 agreement coefficient of 0.751-0.692 when using a six- and four-level reporting structure, respectively, which translates into "substantial" and "moderate agreement" for a benchmarked verbal equivalent scale. The reported performance metrics are further compared against past forensic footwear reliability studies, including a discussion of how the use of a six-level reporting structure impacts results.

Forensic Footwear Reliability: Part II-Range of Conclusions, Accuracy, and Consensus.

Between February 2017 and August 2018, West Virginia University conducted a reliability study to determine expert performance among forensic footwear examiners in the United States. Throughout the study's duration, 70 examiners each performed 12 comparisons and reported a total of 840 conclusions. In order to assess the accuracy of conclusions, the similarities and differences between mated and nonmated pairs were evaluated according to three criteria: (i) inherent agreement/disagreement in class, wear, and randomly acquired features, (ii) limitations as a function of questioned impression quality, clarity, and totality, and (iii) adherence to the Scientific Working Group for Shoeprint and Tire Tread Evidence (SWGTREAD) 2013 conclusion standard. Using these criteria, acceptable/expected categorical conclusions were defined. Preliminary results from this study are divided into a series of three summaries. This manuscript (Part II) reports accuracy and reproducibility. For mated pairs, accuracy equals 76.3% ± 13.0% (median of 78.6% and a 90% confidence interval between 72.2% and 80.0%). For nonmated pairs, accuracy equals 87.4% ± 9.24% (median of 91.4% and a 90% confidence interval between 84.7% and 89.8%). In addition, the community assessed agreement (denoted by IQR) of reported results equals the research team's accepted/expected conclusions for 10 out of 12 comparisons. In terms of reproducibility, the 90% confidence interval for consensus was computed and found to equal 0.71-0.86 (median of 0.77) for the combined dataset. Although based on a limited sample size, these results provide a baseline estimate of accuracy and consensus/reproducibility as a function of the existing seven-point SWGTREAD 2013 conclusion standard.

Forensic Footwear Reliability: Part I-Participant Demographics and Examiner Agreement.

In order to assess the extent of agreement between forensic footwear examiners in the United States, a reliability study was performed by West Virginia University between February 2017 and August 2018. Over the span of 19 months, 70 examiners each performed 12 comparisons and reported a total of 840 conclusions. For each comparison, participants were queried on a number of factors in order to determine the degree to which different types of features were identified, evaluated, and weighted, before arriving at a final decision regarding the strength of the association or disassociation between questioned and test impressions. Preliminary results from this study are divided into a series of three summaries. This manuscript (Part I) describes participant demographics as well as community agreement in both feature identification/annotation, and final reporting. Results indicate considerable variation in feature identification/annotation (as low as 66.5% agreement), but higher consistency in the reporting of overall conclusions. For mated pairs, this agreement was 79.7% ± 14.1% (median of 85.7% and a 90% confidence interval between 75.9% and 83.2%). For nonmated pairs, the equivalent overall agreement was 89.8% ± 6.69% (median of 91.4% and a 90% confidence interval between 87.4% and 92.1%). These estimates of agreement are further compared with previous published findings, and collectively, the work extends the body of knowledge concerning reliability in forensic footwear comparisons and conclusions.

Empirically observed and predicted estimates of chance association: Estimating the chance association of randomly acquired characteristics in footwear comparisons.

The power associated with demonstrating a linkage between footwear and an impression left at the scene of a crime is directly related to the perceived rarity of the shoeprint itself. When individualizing characteristics are present, their relative position, orientation, size and shape are examined and compared with known exemplars to establish the strength of the suspected linkage. However, the degree to which a feature might repeat by chance alone is less well documented in many pattern science fields, including forensic footwear analysis. To assess this chance, the mathematical similarity of more than 3.2 million pairwise RAC comparisons was performed, as a function of more than 72,000 RACs, collected from 1300 unrelated outsoles. The resulting similarity scores were sorted, and more than 91,000 of the mathematically most similar known non-match RACs with positional co-occurrence were visually assessed (in duplicate by two analysts) to determine their degree of observable resemblance. These empirical estimates of visual similarity were used to model resemblance as a Bernoulli distribution with binary outcomes (indistinguishable/distinguishable). Using a logistic regression, the conditional probability of sufficient resemblance, given a mathematical match score, or p(indistinguishability|score), was estimated and used to predict the likelihood of encountering indistinguishable features for the remaining less-similar 1.0 million RAC pairs in the dataset with the same geometric/shape categorization (linear, compact or variable) (1, 105, 943-91, 607=1, 014, 336). Part 1 of this effort reports the intersection of co-occurrence in spatial position and resemblance with results indicating that the median estimate of indistinguishability based on the upper 95% credible interval for estimation (or worst-case scenario) is 1 in 444,126 for linear, 1 in 291,111 for compact, and 1 in 880,774 for variable features. Part 2 of this effort will report random match probabilities for the same dataset.

Eye tracking to evaluate evidence recognition in crime scene investigations.

Crime scene analysts are the core of criminal investigations; decisions made at the scene greatly affect the speed of analysis and the quality of conclusions, thereby directly impacting the successful resolution of a case. If an examiner fails to recognize the pertinence of an item on scene, the analyst's theory regarding the crime will be limited. Conversely, unselective evidence collection will most likely include irrelevant material, thus increasing a forensic laboratory's backlog and potentially sending the investigation into an unproductive and costly direction. Therefore, it is critical that analysts recognize and properly evaluate forensic evidence that can assess the relative support of differing hypotheses related to event reconstruction. With this in mind, the aim of this study was to determine if quantitative eye tracking data and qualitative reconstruction accuracy could be used to distinguish investigator expertise. In order to assess this, 32 participants were successfully recruited and categorized as experts or trained novices based on their practical experiences and educational backgrounds. Each volunteer then processed a mock crime scene while wearing a mobile eye tracker, wherein visual fixations, durations, search patterns, and reconstruction accuracy were evaluated. The eye tracking data (dwell time and task percentage on areas of interest or AOIs) were compared using Earth Mover's Distance (EMD) and the Needleman-Wunsch (N-W) algorithm, revealing significant group differences for both search duration (EMD), as well as search sequence (N-W). More specifically, experts exhibited greater dissimilarity in search duration, but greater similarity in search sequences than their novice counterparts. In addition to the quantitative visual assessment of examiner variability, each participant's reconstruction skill was assessed using a 22-point binary scoring system, in which significant group differences were detected as a function of total reconstruction accuracy. This result, coupled with the fact that the study failed to detect a significant difference between the groups when evaluating the total time needed to complete the investigation, indicates that experts are more efficient and effective. Finally, the results presented here provide a basis for continued research in the use of eye trackers to assess expertise in complex and distributed environments, including suggestions for future work, and cautions regarding the degree to which visual attention can infer cognitive understanding.

Classification of footwear outsole patterns using Fourier transform and local interest points.

Successful classification of questioned footwear has tremendous evidentiary value; the result can minimize the potential suspect pool and link a suspect to a victim, a crime scene, or even multiple crime scenes to each other. With this in mind, several different automated and semi-automated classification models have been applied to the forensic footwear recognition problem, with superior performance commonly associated with two different approaches: correlation of image power (magnitude) or phase, and the use of local interest points transformed using the Scale Invariant Feature Transform (SIFT) and compared using Random Sample Consensus (RANSAC). Despite the distinction associated with each of these methods, all three have not been cross-compared using a single dataset, of limited quality (i.e., characteristic of crime scene-like imagery), and created using a wide combination of image inputs. To address this question, the research presented here examines the classification performance of the Fourier-Mellin transform (FMT), phase-only correlation (POC), and local interest points (transformed using SIFT and compared using RANSAC), as a function of inputs that include mixed media (blood and dust), transfer mechanisms (gel lifters), enhancement techniques (digital and chemical) and variations in print substrate (ceramic tiles, vinyl tiles and paper). Results indicate that POC outperforms both FMT and SIFT+RANSAC, regardless of image input (type, quality and totality), and that the difference in stochastic dominance detected for POC is significant across all image comparison scenarios evaluated in this study.

Quantitative assessment of similarity between randomly acquired characteristics on high quality exemplars and crime scene impressions via analysis of feature size and shape.

Forensic footwear evidence can prove invaluable to the resolution of a criminal investigation. Naturally, the value of a comparison varies with the rarity of the evidence, which is a function of both manufactured as well as randomly acquired characteristics (RACs). When focused specifically on the latter of these two types of features, empirical evidence demonstrates high discriminating power for the differentiation of known match and known non-match samples when presented with exemplars of high quality and exhibiting a sufficient number of clear and complex RACs. However, given the dynamic and unpredictable nature of the media, substrate, and deposition process encountered during the commission of a crime, RACs on crime scene prints are expected to exhibit a large range of variability in terms of reproducibility, clarity, and quality. Although the pattern recognition skill of the expert examiner is adept at recognizing and evaluating this type of natural variation, there is little research to suggest that objective and numerical metrics can globally process this variation when presented with RACs from degraded crime scene quality prints. As such, the goal of this study was to mathematically compare the loss and similarity of RACs in high quality exemplars versus crime-scene-like quality impressions as a function of RAC shape, perimeter, area, and common source. Results indicate that the unpredictable conditions associated with crime scene print production promotes RAC loss that varies between 33% and 100% with an average of 85%, and that when the entire outsole is taken as a constellation of features (or a RAC map), 64% of the crime-scene-like impressions exhibited 10 or fewer RACs, resulting in a 0.72 probability of stochastic dominance. Given this, individual RAC description and correspondence were further explored using five simple, but objective, numerical metrics of similarity. Statistically significant differences in similarity scores for RAC shape and size were consistently detected for three of the five metrics (modified phase only correlation, Euclidean distance, and Hausdorff distance). Conversely, a single metric (the matched filter) expressed the least dependence between score and both shape and size. Moreover, for all crime-scene-like RACs with coincidental association in position, the matched filter produced the greatest discrimination potential in sorting known matches and known non-matches. Despite this demonstrated success, numerical metrics of similarity are not without limitations, and the remainder of this work provides commentary on the difficulties associated with using objective metrics when faced with segmentation, incomplete information, and low signal-to-noise ratios.

Quantifying randomly acquired characteristics on outsoles in terms of shape and position.

Footwear evidence has tremendous forensic value; it can focus a criminal investigation, link suspects to scenes, help reconstruct a series of events, or otherwise provide information vital to the successful resolution of a case. When considering the specific utility of a linkage, the strength of the connection between source footwear and an impression left at the scene of a crime varies with the known rarity of the shoeprint itself, which is a function of the class characteristics, as well as the complexity, clarity, and quality of randomly acquired characteristics (RACs) available for analysis. To help elucidate the discrimination potential of footwear as a source of forensic evidence, the aim of this research is to further characterize the chance association in position, shape, and geometry of RACs on a semi-random selection of footwear. To accomplish this goal in an efficient manner, a partially automated image processing chain was required, including steps for automated feature characterization. This technical note details the methods, procedures, and type of results available for subsequent statistical analysis after processing a collection of more than 1000 shoes and 57,426 randomly acquired characteristics.